University Entrance Examinations as a Benchmark Resource for NLP-based Problem Solving
نویسندگان
چکیده
This paper describes a corpus comprised of university entrance examinations, which is aimed to promote research on NLP-based problem solving. Since entrance examinations are created for quantifying human ability of problem solving, they are a desirable resource for benchmarking NLP-based problem solving systems. However, as entrance examinations involve a variety of subjects and types of questions, in order to pursue focused research on specific NLP technologies, it is necessary to break down entire examinations into individual NLP subtasks. For this purpose, we provide annotations of question classifications in terms of answer types and knowledge types. In this paper, we also describe research issues by referring to results of question classification, and introduce two international shared tasks that employed our resource for developing their evaluation data sets.
منابع مشابه
An Efficient Genetic Agorithm for Solving the Multi-Mode Resource-Constrained Project Scheduling Problem Based on Random Key Representation
In this paper, a new genetic algorithm (GA) is presented for solving the multi-mode resource-constrained project scheduling problem (MRCPSP) with minimization of project makespan as the objective subject to resource and precedence constraints. A random key and the related mode list (ML) representation scheme are used as encoding schemes and the multi-mode serial schedule generation scheme (MSSG...
متن کاملOverview of Todai Robot Project and Evaluation Framework of its NLP-based Problem Solving
We introduce the organization of the Todai Robot Project and discuss its achievements. The Todai Robot Project task focuses on benchmarking NLP systems for problem solving. This task encourages NLP-based systems to solve real high-school examinations. We describe the details of the method to manage question resources and their correct answers, answering tools and participation by researchers in...
متن کاملHybrid Meta-heuristic Algorithm for Task Assignment Problem
Task assignment problem (TAP) involves assigning a number of tasks to a number of processors in distributed computing systems and its objective is to minimize the sum of the total execution and communication costs, subject to all of the resource constraints. TAP is a combinatorial optimization problem and NP-complete. This paper proposes a hybrid meta-heuristic algorithm for solving TAP in a ...
متن کاملBasic ratio-based DEA models
In this paper, different input-oriented ratio-based DEA (DEA-R-I) models are proposed. By presenting the envelopment-additive-enhanced Russell model based on the DEA-R-I form, each decision making unit is evaluated and its efficiency score is calculated. Also, by presenting the central resource allocation model based on the DEA-R-I form, a suitable benchmark for all DMUs is proposed by solvin...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
عنوان ژورنال:
دوره شماره
صفحات -
تاریخ انتشار 2013